Characterizing a workload for the comparison of interactive services

Characterizing the workload is a necessary part of any system performance evaluation effort. This necessity is a consequence of the sensitivity of the performance indices of interest, be they productivity or responsiveness indices, to the nature, composition and intensity of the workload processed by the system being evaluated. No matter how we define it, performance cannot be expressed by quantities independent of the system's workload. 1 Similarly, the performance of two or more systems cannot be meaningfully compared if these systems do not process the same workload. This statement applies not only to procurement problems, but also to problems of improvement and design, that is, whenever a performance comparison is to be made between two systems which differ from each other in some respect (see Figure 1). Since the purpose of the comparison is to expose the relative merits of the two systems, configurations, versions, architectures, or solutions through the differences between the values of some suitable performance indices, the workloads they process must be the same if their influences on these values are to be identical. Otherwise, the differences between the indices will reflect those between the workloads as well as those between the systems, and it will be difficult or impossible to distinguish the respective contributions. The evaluation of a system's performance, or the comparison of the performances of several systems, should ideally be made under the workload that the system, or any of the competing systems, will have to process from the prescnt to the end of its lifetime. This is, however, impossible, at least for most installations, since the workload is time-variant and its future evolution unknown. The only exception is that of those installations whose workload is constant in composition and intensity, and perfectly known (e.g., certain industrial control and manufacturing computers). Moreover, even if the whole future workload were known (a very unrealistic assumption in the great majority of the cases), measuring or computing the performance of the system under its total workload would be impossible or impractical. A much more compact workload is therefore to be used in evaluations or comparisons. Since performance indices depend on the workload's nature and composition,